Helical Edge Modes near Transition to Topological Insulator with Indirect Gap 



O 

(N 

> 
O 

00 



Shijun Mao 1,2 and Yoshio Kuramoto 1 
Department of Physics, Tohoku University, Sendai 980-8578, Japan 1 
Department of Physics, Tsinghua University, Beijing 100084, P-R. China 2 
(Dated: November 9, 2010) 

Helical edge modes are characteristic of topological insulators in two dimensions. This paper 
demonstrates that helical edge modes remain across transitions to ordinary insulators or to semimet- 
als under certain condition. Straight and zigzag edges are considered in a tight-binding model on 
square lattice. We focus on the case of indirect gap in bulk topological insulators, and obtain the 
spectrum of edge modes on both sides of transitions. For straight edge, the helical edge mode in 
topological insulators with strong particle-hole asymmetry has a reentrant region in momentum 
space. Edge modes show up even in ordinary insulators, but are absent in semimetals. In zigzag 
edge, the helical edge mode survives in both semimetals and ordinary insulators. However, the edge 
modes are absent inside the energy gap of ordinary insulators. All results are obtained analytically. 
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I. INTRODUCTION 

In recent intensive study of topological insulators (TI) , 
property of helical edge (2D TI) or surface (3D TI) states 
is one of the key topics. It has been shown that these 
states are robust against interaction and disorder [l|, Q ■ 
The dissipationlcss spin current has potential application 
in spintronics. In experiments for topological insulator 
materials, some materials are actually bulk metals, but 
still support topological surface states, such as SbJ^Q, 
Bi o .9iSb o .o9&lg,Bi2_ a! Mn a! Te30 andBi 2 Se 3 0,0. This 
kind of surface modes in (semi) metallic systems deserves 
further study, especially as to their implication to the 
topologically nontrivial and trivial systems. With real- 
istic energy bands for Bi2Sc3, which arc obtained by ab 
initio band calculation, the spectrum of helical surface 
modes has already been derived numerically (To| . How- 
ever, except for the low-energy part, further property of 
the surface mode, such as the existence region in mo- 
mentum space, has not been discussed. Note that the 
topological argument about the robustness of the mode 
applies only to the energy region where bulk excitations 
are absent. In this paper we are interested in global prop- 
erty of the helical modes including the region overlapping 
with bulk excitations. 

One of the widely used models for 2D TI has been pro- 
posed by Bernevig, Hughes and Zhang (lT|. and is called 
the BHZ model. Helical edge states have been further 
studied using the resultant continuum model [lU, [l3[ and 
the lattice regularized model fl4Hll| . In these previous 
papers, the particle-hole symmetry has been assumed. 
Namely, the conduction and valence bands have the same 
spectrum except for the sign. Hence the direct energy 
gap follows. In materials such as Sb and Bio.91Sbo.09, 
however, the energy gap or overlap involves different po- 
sitions in momentum space. Hence theoretical study be- 
yond the particle-hole symmetric case is of practical in- 
terest. 

In this paper, we study these helical states in both 



topologically nontrivial and trivial cases. In order to 
clarify the characteristics associated with the particle- 
hole asymmetry in the simplest manner, we take the BHZ 
model on the square lattice, and derive the spectrum of 
helical modes analytically. We consider a strip geome- 
try of the 2D system with straight and zigzag edges as 
in ref . fl(| . and generalize its method for deriving helical 
edge states to particle-hole asymmetric case. 

The paper is organized as follows: In §2, we review the 
particle-hole asymmetric BHZ tight-binding model, pay- 
ing attention to both direct and indirect gap-closing in 
2D Brillouin zone (BZ). Sections 3 and 4 are devoted to 
the straight edge and zigzag edge systems, respectively. 
We analytically derive spectrum of helical edge states in 
particle-hole asymmetric system. In straight edge case, 
edge mode is present in TI and, under appropriate con- 
ditions, also in ordinary insulators (01). However, it's 
absent in semimetals (SM). In zigzag edge case, helical 
edge states survive in each case of TI, 01, and SM. The 
summary and outlook are given in §5. 



II. PARTICLE-HOLE ASYMMETRIC BHZ 
MODEL 

We consider the BHZ model given by the following 
block diagonalized 4x4 matrix: 



H(k) 



h(k) 
h*(-k) 



(1) 



where k = (k x , k y ) is a 2D crystal momentum. The lower- 
right block h*{—k) for down-spin is deduced from the 
upper-left block h(k) for up-spin by time-reversal trans- 
formation. We parametrize h(k) as 

e + cL d r — id. 



h{k) = e(k)o- + d(k) ■ a 



id,, 



(2) 



where <tq is the 2x2 unit matrix in the pseudo-spin 
space representing the two kinds of orbitals, and a is 
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the vector composed of the Pauli matrices a x , a y and a z . 
We write a = 1 hereafter unless confusion arises. We 
consider the BHZ model over the whole BZ of the square 
lattice. Then the energy parameters are given by the 
tight-binding form as 



e(k) = C -2D(2-cosk x 
d x {k) = Asmk x , d y (k) = 
d z (k) = A - 25(2 - cosfc^ 



— cos ky) 
A sin k y , 

- cos ky), 



(3) 
(4) 
(5) 



with the lattice constant a set to unity 
The bulk energy E^(k) is obtained as 

E h ±(k) =e(k) ± {A 2 (sin 2 k x + sin 2 k y ) 

+ [A -45 + 25 (cosfcz + cosfc y )] 2 } 



1/2 



(0) 



It is clear that finite e(fc) breaks the particle-hole symme- 
try. Since C in Eq.© merely describes the energy shift 
of the origin, we put C = for simplicity. As in our 
previous paperfl6|. we use B as the unit of energy and 
put B = 1 hereafter. 

The change of A will cause transition between TI and 
01. With inversion symmetry, the direct gap can close 
only at time-reversal invariant points, namely at T = 
(0,0), X = (tt,0), X' = (0,tt) and M = {it, it) [H, 
where bulk energies become 



5 b± (0,0)=±|A|, 
£b±(0,7r) = -45 ± | A 
E b± (7r,7r) = -8£)±|A. 



4\ = E h± (n,0), 



(7) 
(8) 
(9) 



As one varies the parameter A, the direct gap closes at 
r, for example, when A = 0. Similarly the gap closes at 
X (also X') with A = 4, and at M with A = 8. 

Due to finite e(k), it is also possible to have the indirect 
gap-closing. For example, we obtain 5 b+ (0,7r) = —A 
with < A < 4 and D = 1 from Eq.©, which becomes 
the same as £7 b _(0, 0) = — A. Then the transition occurs 
between TI and SM at D = 1 for < A < 4. 

In this paper we only deal with edge mode with up- 
spin and confine to the case A, D > and A < 4, since 
the down-spin part is obtained by using the time-reversal 
symmetry. Note that the result for negative D is obtained 
by changing the role of conduction and valence bands. 
On the other hand, the case A > 4 is understood by 
changing the role of T and M points in the Brillouin 
zone [la]. 



III. STRAIGHT EDGE 



Analytic solution of spectrum and wave 
function 



Let us first consider a straight edge geometry (see Fig. 
|TJl) , in which electrons are confined to N r rows in a 




FIG. 1. (Color online) (a) Straight edge system with bound- 
aries in (1,0) direction. (b)Zigzag edge system with bound- 
aries in (1,1) direction. 



strip between y = 1 and y = N r , i.e., the two edges 
are along x-axis. Translational invariance along x-axis 
allows for constructing a ID tight-binding Hamiltonian 
hoi(k) with k = k x in terms of the creation Cj(fc) and 
annihilation cj(fc) operators with row index J, which are 
two-component vectors in the pseudo-spin space. For up 
spin, the ID model is obtained as 



hf = ^/loi(fc), 
fc 

h 01 {k) = Y J ^ J {k)£{k)cj{k) 

.7 

+ Y1 [cl(k)i y c J+1 (k) + h.c. 



(10) 



(11) 



where £{k) and t y are given by 

£(k) = {-AD + 2Dcosk) 
+ A sin ka x + (Ab 



2 cos k) a z 



.A 

ty — ~ 



D. 



(12) 
(13) 



Here and in the following, we use the notation Ab = 
A — 4. The corresponding Schrodingcr equation is given 
by 



f(/c)*j + ^j_i + 4* 



7+1 



E(k)^ u 



(14) 



where $ j is the two-component amplitude with row in- 
dex J. The straight edges along the J = 1 row and 
J = N r row can be implemented by open boundary con- 
dition * = ^! Nr+l = 0. 

We consider the thermodynamic limit N r — > oo, and 
try an edge state solution with property: *,/+i = X^j = 
A J+1 * with |A| < ldKH. Here A is a complex number 
in general. Then Eq. (fT4"|) can be written in the following 
form: 



£{k) + XP + X-%] * = P i(A, k)V = E{k)^>. (15) 



Although Poi(^i fc) is a 2 x 2 matrix, there is at most one 
solution ^> for the helical edge mode in Eq. (TT5"|) . We de- 
compose Pqi (A, k) into the Hermitian part and the rest 
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called the annihilator[16|. The absence of the particle- 
hole symmetry in the present case requires a little more 
complicated procedure than before [lf|. Let us first in- 
troduce a parameter (f> and make the following transfor- 
mation: 



ax = cos <p a x + sm q> a z 
oz = — sin d> o x + cos 4> c z 



(16) 
(17) 



Then, we have 



£(k)=£ +£ 1 a x +£ 3 a z , (18) 

50 = -4D + 2D cos k, (19) 

51 = A sin k cos <f)+ (As + 2 cos k) sin <f>, (20) 
S3 = — ^4 sin fc sin + (As + 2cos/c) cos</>, (21) 

Xtl+X'Hy = ^A(\- \~ 1 )<7 v +cos<j)(\ + X~ x )a z 

+ (A + A" 1 ) (sin0a x + D) . (22) 

Now we decompose -Poi(A, k) in Eq. (fT5)) as 

P01 = H 01 + F i, (23) 

where the Hcrmitian part H m is chosen as combination 
of (To (= 1) and ax terms. The eigenstate ^ of Hqi obeys 
the relation <tx^ = s'f with s = ±1. Furthermore, cr„ 
and ctz terms combine to form the annihilator Fqi [16j . 
Namely we obtain 

Hoi* = B(fc)*, ^01* = 0, (24) 
H 01 =S + S x g x + (A + A- 1 ) (sin^x + D) , (25) 

*bi = i- (A - A" 1 ) (7, tf + [£ 3 + cos0(A + A- 1 )] a z , 

(26) 

In view of E(k) being real, and the parameter A being 
complex in general, we require in i?oi the condition 

(sin(f)ax + D)^ = 0, (27) 

which determines <j> as 

sin^ = -sD. (28) 

Then E(k) of the edge mode reads 

E{k) = sSx + S . (29) 

The condition for Fq\, which is given by Eq. (|26p . to form 
the annihilator a y + isaz reads 

£ 3 + cos0(A + A" 1 ) =- s ^(A-A- 1 ), (30) 

which determines A since <fi has already been fixed by 
Eq. (|28|) . The solutions A = A s ±(fc) are given by 



±v ' 2cos6 + sA ' v ; 



with cos 4> = vl — F> 2 . The dependence on k comes only 
through S3. For complex A, we obtain 



|A S ±| 



_ 2y/l - D 2 - sA 
~ 2V1 -D 2 + ^A' 



(32) 



which does not depend on k, and should be less than 
unity. Therefore, we must choose s = 1 that corresponds 
to the right-going mode. We obtain the spectrum: 



E t (k) = S 1 +S = -DA + Ay/1 - D 2 sinfc. (33) 

In the case of D — 0, the spectrum is reduced to A sink, 
which was already obtained [T^ - [l6j . The constant term 
—DA can be canceled by the chemical potential. Then 
the zero mode occurs at k = 0. Otherwise the zero mode 
has a finite momentum. The two-component wave func- 
tion is given by 



VTTd 



(34) 



apart from the normalization factor. Hereafter we use 
notation X± = X±±, since only the case of s = 1 is rele- 
vant. 

FigE] (a) shows the spectrum of the system including 
both h\ and h±, i.e., two helical modes. The bulk spec- 
trum is obtained by projecting the 2D spectrum for a 
finite but large system into ID BZ of the straight edge. 
The parameter |A±| is shown in FigJ5] (b). The edge 
modes are present only if |A| < 1 is satisfied for both A±. 
It is clear from Figj2jb) that the momentum range for 
the edge mode is split into two; one is localized around 
center of BZ, and the other allows for a reentrant edge 
mode flM Il6j. Furthermore, the two parts have exactly 
the same width for the momentum range. This fact can 
be understood by writing S3 in Eq. ([?!]) as 



S3 - A B \/l - D 2 = AD sin k + 2 y/l - D 2 cos k 



= y/A 2 D 2 + 4(1 - D 2 ) sin(fc + a), 



(35) 



with tana = 2\/l — D 2 / (AD). This expression shows 
that S3 is symmetric about 



k = k c ~ tt/2 — a = arctan 



AD 



2V1-D 2 ' 



(36) 



which becomes 1.18 with the chosen values of A = 1 and 
D = 0.98. Hence, according to Eq. (f3"Tj) . the value of 
A± is also symmetric around k c as seen from Fig(2jb). 
The allowed momentum range for the edge mode with 
|A±| < 1 is also symmetric around k c . 



B. Transition to semimetals 

FigJ3]shows the critical case with D = 1 where the min- 
imum value of conduction band is equal to the maximum 



FIG. 2. (Color online) (a) Energy band E(k) and (b) param- 
eters |A_| (red) and |A+| (black) in straight edge system with 
A = 1.2, A = 1, D — 0.98. Here, parameter |A±| corresponds 
to right-going branch of edge mode. 



FIG. 3. (Color online) Energy band E(k) and parameter 
|A±| in straight edge system with A = 1.2, A = 1, D = 1.0. 
Here, parameter |A±| corresponds to right-going branch of 
edge mode. 



value of valence band: 

E h+ \ min = E b+ (0,7r) = E h+ {ir,n) = -A (37) 

Eh-\max = -Sb— (0,0) = — A. (38) 

For the parameter A±, we obtain 

A ± = - sin k ± Vl + sin 2 k, (39) 

which means A_ < — 1 for < k < n, and A + > 1 for 
— 7r < k < 0. Therefore, the edge mode is absent for any 
momentum in the critical case. 

For D > 1, the bulk system becomes a semimctal. 
FigfJ] shows the spectrum in this case with D = 1.1. 
Because we have sin0 = — D < —1, the edge mode is 
absent. 

Let us summarize the situation with changing D for 
the straight edge: With D = 0, the system is topolog- 
ically nontrivial insulator with particle-hole symmetry. 
For D > 0, the particle-hole symmetry is broken and the 
shape of valence and conduction bands become different. 
As getting close to the critical case with D = 1, the 
spectrum of the edge mode separates into two parts with 
the same width in the ID momentum space, and both 
parts shrink simultaneously with further increase of D. 
At D = 1, associated with closing of the bulk band gap, 
edge modes disappear in the whole BZ. With D > 1, the 
system enters into semimetals where no edge modes are 
present for the straight edge. 
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FIG. 4. (Color online) Energy band E(k) of straight edge 
system with A = 1.2, A = 1, D = 1.1. 



C. Edge mode in ordinary insulators 

We consider the edge mode in ordinary (topologically 
trivial) insulators (OI). In the par ticle-hole symmetric 
system, there is no edge modes [ill [Til ]. In the present 
case, edge modes can show up even in OI depending on 
the value of D. The presence of edge modes in OI can be 
analyzed conveniently by considering two limiting cases: 
(i) particle-hole symmetric case with D = 0, where we 
obtain £ 3 = Ag + 2 cos A; ; (ii) critical case with D = 1, 
where the bulk energy gap closes and edge mode disap- 
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FIG. 5. (Color online) |A±(fc c )| as a function of D, with 
A = —1.2 and A = 1. Helical edge modes appear for D c \ < 
D < 1 although system is an ordinary insulator. Especially, 
for D C 2 < D < 1, helical edge mode contains two separate 
parts. We obtain D c i = 0.93 and D C 2 = 0.99 with present 
choice of parameters. 



pears. Here we obtain £3 = A sin k. In order to derive the 
momentum region allowing for edge modes, we consider 
extrema of A± at k = k c . We obtain from Eq. ([55|) 

£ 3 = A B Vl-D 2 + ^A 2 D 2 + 4(1 - D 2 ), (40) 
which is negative for 

/ A 2 D 2 \ 1/2 
Ab< -{— ■ (41) 

In the case of £ 3 < we obtain |A+(fc c )| > |A_(fc c )|, and 
otherwise opposite inequality. 

FigEl plots the parameters |A±(fc c )| given by Eq. (j3l"j) 
as a function of D. We set A = —1.2, or Ab = —5.2, 
which makes the system OI. Helical edge modes appear 
for D c i < D < 1 where the boundary D c \ are given by 
solution of the condition 

A+(fc c ) = 1. (42) 

With D increasing from zero to unity, |A+(fc c )| decreases, 
as seen in FigO As long as D < D c \ , we have the relation 
|A + (fc c )| > 1, and no edge modes. 

At D — D c i, we obtain |A_(fc c )| < |A + (fc c )| = 1, and 
the edge mode begins to show up. FiglB] shows an exam- 
ple of the spectrum in the OI with A = — 1.2 and A = 1. 
Here edge modes appear in the middle of the BZ without 
the reentrant behavior. 

With further increase of D, we come close to the lim- 
iting case (ii), and have the reversed relation |A+(fc c )| < 
|A_(fc c )|. At D = D c2 , we obtain |A_(fc c )| = 1. Then 
the edge mode vanishes at k = k Ci but is still present on 
both sides of k c . Thus, with D C 2 < D < 1, edge mode 
contains two separate parts. FigjT] shows the spectrum 
and |A±| corresponding to this case. The separate mo- 
mentum regions, which are symmetric around k c , shrink 
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FIG. 6. (Color online) Conduction band E(k) and parameter 
|A±| for D = 0.96, A = -1.2, A = 1. The parameter |A±| 
corresponds to right-going branch of edge mode, which does 
not have the reentrant behavior. 
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FIG. 7. (Color online) Conduction band E(k) and parameter 
|A±| for D = 0.991. Other parameters are the same as those 
in Fig[6] Reentrant modes are present in this case. 
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simultaneously until D = 1 is reached, where edge modes 
disappear completely. In FigJ7]we obtain k c = 1.307 from 
Eq.(pf. 



IV. ZIGZAG EDGE 

A. Analytic solution 

Let us now consider the zigzag edge geometry, as il- 
lustrated in FigOJb). Electrons are confined in a diag- 
onal strip: 1 < y — x < N r , provided the edges are 
placed at y — x = 1 and y — x = N r , normal to the 
(1, — Indirection. Translational invariancc remains along 
the (1, Indirection. Bulk energy E\, in terms of variables 
K = (k x + k y )/2 and £ = (k x — k y )/2 is obtained from 
Eq.© as 

Eb±(K,Q = — 4D + 4L>cosAtcos£ 

± [2A 2 (sin 2 At cos 2 £ + cos 2 At sin 2 £) 



(Ab + 4 cos k cos 



1/2 



(43) 



The Schrodinger equation for the edge mode analogous 
to Eq.([T3]) reads: 



+ tll^-l + = £ t ( K ) $ j: ( 44 ) 



where 



fn = -4D + A fl tr„ 
in 



Sin AC (Tx — i— p COS AC (Ty , 
v2 



v/2 

2 cos k a z + 2D cos At, 



and we have introduced 

1 / \ 1 , 

a x = -j= {cr x + (T y ) ay = -j= [a y - a x 



(45) 
(46) 

(47) 



Here the conserved momentum is At — (k x + k y )/2, with 
— 7r/2 < n < tt/2. We impose the boundary condition: 
&o = &N r +i = for zigzag edge geometry, and consider 
the thermodynamic limit N r — > oo. 

Assuming eigenstate of Eq. (|44| with property $j = 
= A J $, where |A| < 1, we obtain 



£u +W u (k) +\-Hu(k) $ = Pii(A,k)$ = ^ t (k)$ 



(48) 



In order to construct the annihilator Fn for the zigzag 
edge, we introduce further transformation: 



(J0 X = cos u ax + sin a a z , 
crgz = — sin 9 ax + cos 9 a z . 



(49) 
(50) 



Then, we have 



£ n = -4£> + A B (cr e:r sin (9 + o- 9z cos 9) , (51) 

/ A \ A 

in = [~~f^ sm K cos ^ + 2 cos At sin 6H erga; — i—j=a y cos At 

^4 \ 

— sin At sin6> + 2 cos ft cos 9 ) an, + 2D cos At. 

V2 / 

(52) 



We decompose Pn in Eq. (|48|) as P\\ = H\\ + -Fii where 

H n = (A B sm9a 0x -4D) 

+ (A + A -1 ) (71 aa x + 2D cos At) , (53) 
-F11 = 72 o-y + 73 cr 6z , (54) 



with 



.4 



71 = — — sin At cos 9 + 2 cos At sin I 

v2 

72 = —i—/= cos At (A -1 — A) , 

v2 

73 = As cos 6* 



(55) 
(56) 



+ (A + A *) ^— — ^= sin At sin 9 + 2 cos At cos 9 J . (57) 

Note that Hn contains non-Hcrmitian part due to the 
complex parameter A. Since Hn should have real 
eigenenergy Ef(n), we require the complex part in Hn 
to be ineffective: 

(71 o Bx + 2D cos At) $ = 0, (58) 

with ag x & — s<E>. Then the parameter 9 is determined as 

_ -8sD - sgn K\J A 4 tan 4 k + 8A 2 (1 - D 2 ) tan 2 At 



A 2 tan 2 At 



cos (9 = VI - sin 2 9 (> 0). 
The eigenenergy is given by 

E t (n) = sA B sin6»-4_D, 



(59) 
(60) 

(61) 



where s = +1 should be chosen for the edge mode, as 
shown below. From Eq. (|59j) we obtain the limiting be- 
havior: 



sin (7 -> 



—sgn At (At — > ±7r/2) 
-s£>, («->0) 



(62) 



which gives Bf(0) = —DA and E^(jt/2) = -A+4(l-D). 

The corresponding annihilator fn in Eq. (|54j) should 
be proportional to a y + isag z , which requires 

A B cos 6» + (A + A -1 ) R = s—j= cos At (A -1 - A) . (63) 

v2 
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with 



R = — sin re sin0 + 2 cos re cos 0. (64) 

y/2 



The solution is given by 



A s± (re) 



-A B cos 9 ± cos 2 6> + 2A 2 cos 2 re - AR 2 



2R + sV%A cos / 



which leads, for complex A, to 
|A,±M| 2 = 



2 2i2 — si/2Acosk 



2R + sV2A cos / 



(65) 



(66) 



In order to choose the appropriate branch out of s = ±1, 
we put re = and obtain R = 2 cos 9 > 0. In this case 
the quantity 

|A S± | 2 = (4 cos 0- sV2A)/{A COS0 + 

is less than unity only with s = +1. 

We now consider the case re = ±7r/2. From Eq. (|6~4l we 
obtain R = A/y/2, and then 



A s± (±7r/2) = ±i. 



(67) 



Since A s ±(±7r/2) is complex, we obtain |Ai±| < 1 near 
the zone boundary. This property means that edge mode 
near zone boundary is stable against transition from TI 
to SM (or 01). However, its existent region shrinks with 
increasing D. Hereafter, we use notation A± = Ai±(re). 



B. Edge modes in TI and topologically trivial cases 

In this subsection wefix A = ±1.2,A= 1, which real- 
izes TI with < D < 1 and A > 0. Note that re = is 
always a crossing point of two helical modes, as a result of 
time-reversal symmetry of the system. Hence this prop- 
erty is robust against particle-hole symmetry breaking 
as long as the system is still in TI. However, the existent 
regions of edge mode will shrink with increasing D. 

FigE] shows the spectrum and |A±| in the case of D = 
0.9, A = 1.2. As shown in the lower panel, we always have 
|A±| < 1 for < re < it/2, which means the presence of 
the edge mode in the whole region of positive k. With 
re < 0, there appears a reentrant edge mode near the zone 
boundary re = — 7r/2, which is not seen in the upper panel 
because of its much lower energy. The binding energy of 
reentrant edge mode near zone boundary is expanded as 



E t -E h 



-4A, 



D + l 
A 



7T 

K+ 2 



(68) 



The reentrant mode merges with bulk excitations at mo- 
mentum K m that is given by the solution of 

(As + 4 cos re) cos 9-V2A sin re sin U 
= A B cos0 + 2i? = 0. (69) 




FIG. 8. (Color online) Energy spectrum E(k) and parameters 
|A±| in zigzag edge system with A — 1.2, A = 1, D = 0.9. 



The analytic expression of n m in terms of system param- 
eters is complicated and it's not illuminating. For the 
parameters used in FigJSl we obtain the numerical value: 
K m = — 0.9367r/2. Since n m is close to — 7r/2, the binding 
energy is only of the order of 10~ 3 according to Eq. (|B5| . 
The reentrant mode vanishes also at the zone boundary 
because of the relation: |A±| — > 1 as re — > — ir/2. 

In critical case with D = 1, the bulk energy gap closes, 
with extremum of given by 

E h+ \ min = E h+ (ir/2, tt/2) = - A (70) 
E h -\ max = £b_(0,0) = -A (71) 

FiglHl shows the spectrum and parameters |A±| in this 
case. Since we obtain |A±(re = 0)| = 1 with D = 1, the 
crossing point of two helical edge modes merges with bulk 
excitations associated with indirect gap-closing, How- 
ever, edge mode still exists in the other momentum region 
with |A±| < 1. Note that the right-going helical mode 
has the flat spectrum for re > 0, and that the left-going 
helical mode has the flat spectrum for re < 0, 

With D > 1, the system is no longer a topological in- 
sulator. However the edge mode still survives in some 
momentum region. Fig llOl shows the spectrum and pa- 
rameters |A±| in this case. The edge mode contains two 
separate parts lying near the zone boundary in valence 
and conduction bands. This new phase is bulk metal 
but with helical edge mode. Note that the solution sin 
becomes complex in certain momentum region, which is 
unphysical. 



8 





-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 
K 




-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 
K 



FIG. 10. (Color online) The same quantities as in Figs[8]and 
[9] except for D = 1.1 in this case. The dashed lines in the 
lower panel correspond to complex sin (9 which is unphysical. 



Let us now consider the OI case with A = —1.2. 
According to Eq. (|67|) . edge mode will appear near the 
boundary of BZ. FigfTT](a) shows an example of the spec- 
trum (upper panel). There is no edge mode within the 
energy gap. FigQj] (b) shows the absence of the edge 
mode at k = 0, since at least one of the |A±| is larger 
than unity. 

V. SUMMARY 

In this paper, we have analyzed edge modes in 
the particle-hole asymmetric BHZ model taking both 
straight and zigzag edges. Our particular attention has 
been the survival or disappearance of helical edge modes 
through transition from TI to SM, and also its presence 
in OI. 

For straight edge, with increasing particle-hole asym- 
metry controlled by D, the edge mode breaks up into 
two parts in momentum space, and each part shrinks si- 
multaneously. Associated with transition from TI to SM, 
the system becomes normal SM without edge mode. On 
the other hand, in zigzag edge case, helical edge mode is 
robust against particle-hole symmetry breaking. In TI, 
gapless point always lies at the T point and reentrant edge 
mode appears near the boundary of ID BZ. With closing 
of indirect gap in bulk energy band, those edge modes 
near the crossing point of Kramers' pair disappear, while 
the edge mode near the zone boundary remains. 
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In 01 systems, helical edge mode, if exists, should not 
be inside the energy gap. Then such edge mode does 
not change the topology of energy band from 01 without 
edge modes. In straight edge system, the presence of edge 
mode is sensitive to parameters. However, for zigzag edge 
case, it is insensitive to values of parameter because the 
edge mode always merges with bulk modes at the zone 



boundary. 

In summary, we have demonstrated the presence of 
helical edge modes outside the energy gap as a novel state 
in particle-hole asymmetric BHZ model. Our analytic 
approach to study the helical modes can be generalized 
to 3D systems. The results will be reported elsewhere. 
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